COVID-19 Impact on Family Medicine Residents Exam Performance.

BACKGROUND AND OBJECTIVES: The COVID-19 pandemic began interrupting family medicine residency training in spring 2020. While a decline in scores on the American Board of Family Medicine In-Training Examination (ITE) has been observed, whether this decline has translated into the high-stakes Family Medicine Certification Examination (FMCE) is unclear. The goal of this study was to systematically assess the magnitude of COVID-19 impact on medical knowledge acquisition during residency, as measured by the ITE and FMCE. METHODS: A total of 19,101 initial certification candidates from 2017 to 2022 were included in this study. Annual ITE scores and FMCE scores were reported on the same scale (200-800) and served as the outcome measure. We conducted multilevel regression analysis to determine ITE score growth and FMCE scores compared to cohorts prior to COVID-19. RESULTS: During COVID-19, the increase in ITE scores from postgraduate year 2 (PGY-2) to PGY-3 was 25.5 points less, representing a 57.6% relative decrease; and from PGY-3 ITE to FMCE, it was 8.6 points less, a 12.7% relative decrease, compared with cohorts prior to COVID-19. FMCE scores were 6.6 points less during COVID-19, representing a 1.2% relative decline from the average FMCE score prior to COVID-19. CONCLUSIONS: This study found nonsubstantive COVID-19 impact on FMCE scores, but a considerable knowledge acquisition decline during residency, especially during the PGY-2 to PGY-3 period. While COVID-19 impacted learning, our findings indicated that residencies were largely able to remediate knowledge deficits before residents took the FMCE.

Differences in Physician Performance and Self-rated Confidence on High- and Low-Stakes Knowledge Assessments in Board Certification.

INTRODUCTION: Evidence links assessment to optimal learning, affirming that physicians are more likely to study, learn, and practice skills when some form of consequence ("stakes") may result from an assessment. We lack evidence, however, on how physicians' confidence in their knowledge relates to performance on assessments, and whether this varies based on the stakes of the assessment. METHODS: Our retrospective repeated-measures design compared differences in patterns of physician answer accuracy and answer confidence among physicians participating in both a high-stakes and a low-stakes longitudinal assessment of the American Board of Family Medicine. RESULTS: After 1 and 2 years, participants were more often correct but less confident in their accuracy on a higher-stakes longitudinal knowledge assessment compared with a lower-stakes assessment. There were no differences in question difficulty between the two platforms. Variation existed between platforms in time spent answering questions, use of resources to answer questions, and perceived question relevance to practice. DISCUSSION: This novel study of physician certification suggests that the accuracy of physician performance increases with higher stakes, even as self-reported confidence in their knowledge declines. It suggests that physicians may be more engaged in higher-stakes compared with lower-stakes assessments. With medical knowledge growing exponentially, these analyses provide an example of the complementary roles of higher- and lower-stakes knowledge assessment in supporting physician learning during continuing specialty board certification.

Racial/Ethnic Group Trajectory Differences in Exam Performance Among US Family Medicine Residents.

BACKGROUND AND OBJECTIVES: Racial/ethnic score disparities on standardized tests are well documented. Such differences on the American Board of Family Medicine (ABFM) certification examination have not been previously reported. If such differences exist, it could be due to differences in knowledge at the beginning of residency or due to variations in the rate of knowledge acquisition during residency. Our objective was to examine the residents' mean initial scores and score trajectories using the In-Training Examination (ITE) and certification examination. METHODS: A total of 17,275 certification candidates from 2014 to 2019 were included in this study. Annual ITE scores and certification examination scores are reported on the same scale and serve as the outcome. We conducted multilevel longitudinal regression to determine initial knowledge and growth in knowledge acquisition during residency by race/ethnicity categories. RESULTS: The mean postgraduate year 1 (PGY-1) ITE score was 393.3, with minority residents scoring 16.2 to 36.0 points lower compared to White residents. The mean increase per year in exam performance from PGY-1 ITE to the certification exam was 39.9 points (95% CI, 38.7, 41.1) with additional change among race/ethnicity categories per year of -3.2 to 1.9 points. CONCLUSIONS: This study found that there were initial score disparities across race/ethnicity groups in PGY-1, and these disparities continued at the same rate throughout residency training, suggesting equality in acquisition of knowledge during family medicine residency training but with a persistent gap throughout training.

Racial/Ethnic Representation Among American Board of Family Medicine Certification Candidates from 1970 to 2020.

BACKGROUND: Because improved patient outcomes and experiences have been associated with health care workforce diversity, efforts to create a diverse family physician workforce have increased. However, a metric that could properly measure family physician representation in various contexts has seldom been studied. OBJECTIVE: The goal of this study is to propose a new metric logRQ and use it to examine the diversification progress of American Board of Family Medicine (ABFM) certification candidates relative to national, state, and historic populations, as well as medical school matriculants. METHODS: We obtained race/ethnicity for certification candidates from the 2014 to 2020 ABFM Certification Examination Registration questionnaire and examined racial/ethnic representation relative to various populations via logRQs. RESULTS: The total sample comprised 26,368 initial certification candidates and 55,347 continuing certification candidates. Asian, Hispanic, and Black's logRQ increased by 0.51, 0.42, and 0.41, respectively, in initial certification candidates compared with continuing certification candidates. In addition, logRQ standard deviation ranged from 0.19 to 0.87 across States, indicating state-level variation. Although Black and Hispanic remained underrepresented, the degree of underrepresentation had improved substantially across the past 5 decades, with logRQ increasing from -2.12 (Black) and -1.16 (Hispanic) in the 1970s to -0.46 (Black) and -0.68 (Hispanic) in the 2010s. The race/ethnicity logRQs of 2020 initial certification candidates relative to 2013 to 2014 medical school matriculants were all near 0, reflecting equitable representation. CONCLUSION: We utilized the proposed metric logRQ to quantify the advancement in representation among ABFM certification candidates in different contexts. The proposed logRQ may serve as a useful tool to monitor representation progress systematically.

The American Board of Family Medicine's 8 Years of Experience with Differential Item Functioning.

INTRODUCTION: Differential item functioning (DIF) procedures flag examination questions in which examinees from different subpopulations who are of equal ability do not have the same probability of answering it correctly. Few medical certification boards employ DIF procedures because they do not collect the needed data on the examinee's race or ethnicity. This article summarizes the American Board of Family Medicine's (ABFM) combined use of DIF procedures and an expert panel to review certification questions for bias. METHODS: ABFM certification examination data from 2013 to 2020 were analyzed using a DIF procedure to flag questions with possible ethnic or racial bias. The flagged questions were reviewed by a racially and ethnically diverse panel of content experts. If the panel judged the source of the DIF was not clinically relevant for the practice of family medicine, the question was removed from the examination. RESULTS: Out of the 3487 questions analyzed, 374 unique questions (11%) were flagged by DIF procedures as potentially biased. Of the flagged questions, the review panel felt 4 should be removed for fairness. DISCUSSION: Using DIF procedures and panel review can improve the quality of the board certification questions and demonstrate the organization's commitment to avoid racial or ethnic bias.

Validating the Test Plan Specifications for the American Board of Family Medicine's Certification Examination.

PURPOSE: To demonstrate the degree to which the American Board of Family Medicine's certification examination is representative of family physician practice with regard to frequency of diagnoses encounter and the criticality of the diagnoses. METHODS: Data from 2012 National Ambulatory Medical Care Survey was used to assess the frequency of diagnoses encountered by family physicians nationally. These diagnoses were also rated by a panel of content experts for how critical it was to diagnose and treat the condition correctly and then assign the condition to 1 of the 16 content categories used on the American Board of Family Medicine examination. These ratings of frequency and criticality were used to create 7 different new schemas to compute percentages for the content categories. RESULTS: The content category percentages for the 7 different schemas correlated with the 2006 to 2016 test plan percentages from 0.50 to 0.90 with the frequency conditions being more highly correlated and the criticality conditions being less correlated. CONCLUSIONS: This study supports the continued use of the current Family Medicine Certification Examination content specifications as being representative of current family medicine practice; however, small adjustments might be warranted to permit better representation of the criticality of the topics.

The Relationship Between Board Certification and Disciplinary Actions Against Board-Eligible Family Physicians.

PURPOSE: Lack of specialty board certification has been reported as a significant physician-level predictor of receiving a disciplinary action from a state medical board. This study investigated the association between family physicians receiving a disciplinary action from a state medical board and certification by the American Board of Family Medicine (ABFM). METHOD: Three datasets were merged and a series of logistic regressions were conducted examining the relationship between certification status and disciplinary actions when adjusting for covariates. Data were available from 1976 to 2017. Predictor variables were gender, age, medical training degree type, medical school location, and the severity of the action. RESULTS: Of the family physicians in this sample, 95% (114,454/120,443) had never received any disciplinary action. Having ever been certified was associated with a reduced likelihood of ever receiving an action (odds ratio [OR] = 0.35; 95% confidence interval [CI] = 0.30, 0.40; P < .001), and having held a prior but not current certification at the time of the action was associated with an increase in receiving the most severe type of action (OR = 3.71; 95% CI = 2.24, 6.13; P < .001). CONCLUSIONS: Disciplinary actions are uncommon events. Family physicians who had ever been ABFM certified were less likely to receive an action. The most severe actions were associated with decreased odds of being board certified at the time of the action. Receiving the most severe action type increased the likelihood of physicians holding a prior but not current certification.

Family Medicine Residents' Debt and Certification Examination Performance.

INTRODUCTION: Previous studies have found that medical students and internal medicine residents with high educational debt perform less well on examinations. The purpose of this study was to examine the relationship between educational debt and family medicine residents' performance on initial in-training and board certification examinations. METHODS: Our study was a cross-sectional secondary analysis of American Board of Family Medicine (ABFM) data collected from residents (N=5,828) who registered for the Family Medicine Certification Examination (FMCE) in 2014 and 2015, representing 85.8% of graduating family medicine residents in the United States in those years. Multivariable linear and logistic regression modeling was used to examine the relationship between debt level and examination scores, and also to explore the relationship between debt level and passing the initial FMCE. RESULTS: After controlling for demographic variables, residents with high debt ($150,000 to $249,999) and very high debt (more than $250,000) performed significantly worse than those with no debt on the initial in-training examination (score differences of 14.2 [CI 8.6, 19.8] and 15.8 [CI 10.3, 21.4] points, respectively) and FMCE (score differences of 19.3 points [CI 13.4, 25.3] and 30.4 points [CI 24.6, 36.3], respectively). Additionally, those with debt above $250,000 had half the odds of passing their initial FMCE (OR 0.45; CI 0.27-0.75). CONCLUSIONS: High educational debt is associated with lower examination performance among family medicine residents. This may be because residents with more debt have more stress or fewer day-to-day financial resources. However, confounding factors may also contribute to this association.

Comparative Analysis of the American Board of Family Medicine and American College of Osteopathic Family Physicians In-Training Examinations.

BACKGROUND AND OBJECTIVES: Family medicine residency programs accredited by the Accreditation Council for Graduate Medical Education and the American Osteopathic Association typically require their residents to take the American Board of Family Medicine's In-Training Examination (ITE) and the American College of Osteopathic Family Physicians' In-Service Examination (ISE). With implementation of the single accreditation system (SAS), is it necessary to administer both examinations? This pilot study assessed whether the degree of similarity for the construct of family medicine knowledge and clinical decision making as measured by both exams is high enough to be considered equivalent and analyzed resident ability distribution on both exams. METHODS: A repeated measures design was used to determine how similar and how different the rankings of PGY-3s were with regard to their knowledge of family medicine as measured by the ISE and ITE. Eighteen third-year osteopathic residents participated in the analysis, and the response rate was 100%. RESULTS: The correlation between ISE and ITE rankings was moderately high and significantly different from zero (rs=.76, P<0.05). A Wilcoxon signed rank test indicated that the median ISE score of 62 was not statistically significantly different than the median ITE score of 71 (Z=-0.74, P=0.46, 2-tailed). CONCLUSIONS: The lack of a difference on statistical analysis of ISE scores and the ITE scores of the PGY-3 residents suggests that the cohort of osteopathic residents in family residency programs and the cohort of residents in ACGME-accredited programs seem to be of comparable ability, therefore there is no clear justification for administering both examinations.

Frequency and Criticality of Diagnoses in Family Medicine Practices: From the National Ambulatory Medical Care Survey (NAMCS).

BACKGROUND: Family medicine is a specialty of breadth, providing comprehensive health care for the individual and the family that integrates the broad scope of clinical, social, and behavioral sciences. As such, the scope of practice (SOP) for family medicine is extensive; however, over time many family physicians narrow their SOP. We sought to provide a nationally representative description of the most common and the most critical diagnoses that family physicians see in their practice. METHODS: Data were extracted from the 2012 National Ambulatory Medical Care Survey (NAMCS) to select all ICD-9 codes reported by family physicians. A panel of family physicians then reviewed 1893 ICD-9 codes to place each code into an American Board of Family Medicine Family Medicine Certification Examination test plan specifications (TPS) category and provide a rating for an Index of Harm (IoH). RESULTS: An analysis of all 1893 ICD-9 codes seen by family physicians in the 2012 NAMCS found that 198 ICD-9 codes could not be assigned a TPS category, leaving 1695 ICD-9 codes in the dataset. Top 10 lists of ICD-9 codes by TPS category were created for both frequency and IoH. CONCLUSIONS: This study provides a nationally representative description of the most common diagnoses that family physicians are seeing in their practice and the criticality of these diagnoses. These results provide insight into the domain of the specialty of family medicine. Medical educators may use these results to better tailor education and training to practice.

The Single Graduate Medical Education (GME) Accreditation System Will Change the Future of the Family Medicine Workforce.

BACKGROUND: Due to the Accreditation Council for Graduate Medical Education (ACGME)/American Osteopathic Association (AOA) single-accreditation model, the specialty of family medicine may see as many as 150 programs and 500 trainees in AOA-accredited programs seek ACGME accreditation. This analysis serves to better understand the composition of physicians completing family medicine residency training and their subsequent certification by the American Board of Family Medicine. METHODS: We identified residents who completed an ACGME-accredited or dual-accredited family medicine residency program between 2006 and 2016 and cross-tabulated the data by graduation year and by educational background (US Medical Graduate-MD [USMG-MD], USMG-DO, or International Medical Graduate-MD [IMG-MD]) to examine the cohort composition trend over time. RESULTS: The number and proportion of osteopaths completing family medicine residency training continues to rise concurrent with a decline in the number and proportion of IMGs. Take Rates for USMG-MDs and USMG-IMGs seem stable; however, the Take Rate for the USMG-DOs has generally been rising since 2011. CONCLUSIONS: There is a clear change in the composition of graduating trainees entering the family medicine workforce. As the transition to a single accreditation system for graduate medical education progresses, further shifts in the composition of this workforce should be expected.

Scope of Practice Among Recent Family Medicine Residency Graduates.

BACKGROUND AND OBJECTIVES: The scope of practice among primary care providers varies, and studies have shown that family physicians' scope may be shrinking. We studied the scope of practice among graduates of residencies associated with Preparing the Personal Physician for Practice (P4) and how length of training and individualized education innovations may influence scope. METHODS: We surveyed graduates 18 months after residency between 2008 and 2014. The survey measured self-reported practice characteristics, scope of practice and career satisfaction. We assessed scope using individual practice components (25 clinical activities, 30 procedures) and a scaled score (P4-SOP) that measured breadth of practice scope. We conducted subgroup analyses according to exposure to innovations over the project period and exposure to specific innovations. RESULTS: No significant differences were found in mean P4-SOP scores between the Pre and Full P4 groups. Compared to national data, P4 graduates reported higher rates for vaginal deliveries (19.3% vs 9.2%), adult inpatient care (48.5% vs 33.7%) and nursing home care (25.4 vs 11.7%) in practice. Graduates exposed to innovations that lengthened training, compared to standard training length, were more likely to include adult hospital care (58.2% vs 38.5%, P=0.002), adult ICU care (30.6% vs 19.2%, P=0.047) and newborn resuscitation (25.6% vs 14%, P=0.028) in their practice and performed 19/30 procedures at higher rates. Graduates of programs with individualized training innovations reported no significant differences in scope compared to graduates without this innovation. CONCLUSIONS: Graduates of residencies engaged in significant educational redesign report a broad scope of practice. Innovations around the length of training may broaden scope and individualized education appears not to constrict scope.

Performance of Graduating Residents on the American Board of Family Medicine Certification Examination 2009-2016.

In response to growing concern about the declining performance on the American Board of Family Medicine Certification Examination, several strategies were employed to assist program directors with preparing their residents to take the examination. The effect of these efforts seems to have resulted in significant improvement in performance.

Prevalence of Burnout in Board Certified Family Physicians.

Physician burnout has become a critical issue in a rapidly changing health care environment and is reported to be increasing. However, little is known about the prevalence of this problem among board-certified family physicians. Using an abbreviated burnout survey, we found a lower prevalence of this problem than has been previously reported.